Summarizing salient events in unmanned aerial videos

ABSTRACT

A method for summarizing image content from video images received from a moving camera includes detecting foreground objects in the images, determining moving objects of interest from the foreground objects, tracking the moving objects, rating movements of the tracked objects, and generating a list of highly rated segments within the video images based on the ratings.

BACKGROUND

1. Technical Field

The present disclosure relates to the field of unmanned aerial vehicles,and more particularly to presentation of events or objects captured bythe vehicles.

2. Discussion of Related Art

An unmanned aerial vehicle (UAV), commonly known as a drone is anaircraft without a human pilot on board. Its flight is either controlledautonomously by computers in the vehicles, or under the remote controlof a pilot on the ground or in another vehicle. UAVs are predominantlydeployed for military applications, but are also used in a small butgrowing number of civil applications, such surveillance of pipelines,natural resource exploration, livestock monitoring, wildfire mapping,transport of medical supplies, etc.

A UAV may be fitted with one or more high definition cameras that cansend streaming video to a remote computer for storage thereon. Anoperator of the remote computer can then review the stored video todetermine whether any events of interest have occurred. For example, ifthe UAV is being used by police to investigate a drug smugglingoperation, an event of interest might include images of armed peoplemoving containers into a suspected drug storage facility. While theentire video captured by the UAV may include several hours or days worthof footage, the event of interest may only comprise a few minutes. Thus,it could take an operator manually reviewing the video several hours todetermine whether the event of interest has occurred. Further, since theevent of interest may occur in a short period of time, the operator caneasily miss the event.

Accordingly, there is a need for methods and systems that canautomatically summarize the events of interest.

BRIEF SUMMARY

According to an exemplary embodiment of the invention, a method forsummarizing image content from video images received from a movingcamera includes: detecting foreground objects in the images, determiningmoving objects of interest from the foreground objects, tracking themoving objects, rating movements of the tracked objects, and generatinga list of highly rated segments within the video images based on theratings.

According to an exemplary embodiment of the invention, a method ofpresenting a summary of a video includes: tracking moving objects withinthe video, determining a smoothness value of a trajectory of eachtracked object, discarding the tracked objects having smoothness valuesbelow a minimum threshold smoothness level, selecting a period of timewithin the video that includes at least one of the remaining trackedobjects, and presenting a frame of the video overlaid with all of theremaining tracked objects that are present during the selected period oftime.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Exemplary embodiments of the invention can be understood in more detailfrom the following descriptions taken in conjunction with theaccompanying drawings in which:

FIG. 1 illustrates a system that operates in accordance with anexemplary embodiment of the present invention.

FIG. 2 illustrates a method of generating a summary visualizationaccording to an exemplary embodiment of the invention.

FIG. 3 illustrates an example of the summary visualization.

FIG. 4 illustrates a method of detecting objects according to anexemplary embodiment of the invention that may be used in generating thesummary visualization.

FIG. 5 illustrates an exemplary trajectory of one of the detectedobjects.

FIG. 6 illustrates further exemplary trajectories of the detectedobjects along with a saliency metric.

FIG. 7 illustrates an example of a user interface that can be used topresent the summary visualization.

FIG. 8 illustrates a method of generating a summary visualizationaccording to an exemplary embodiment of the invention.

FIG. 9 illustrates another example of a user interface that can be usedto present the summary visualization.

FIG. 10 illustrates an example of a computer system capable ofimplementing methods and systems according to embodiments of thedisclosure.

DETAILED DESCRIPTION

Embodiments of the present invention relates to methods and/or systemsthat may be used to provide a summary of events in unmanned aerialvideos.

Referring to FIG. 1, there is illustrated a system that operates inaccordance with an embodiment of the present invention. The systemincludes at least one unmanned aerial vehicle (UAV) 110 and a server140. The UAV 110 includes a processor (not shown), one or more camera(s)130 for capturing video images, and a transmitter 120 for transmittingthe captured video to server 140.

In an exemplary embodiment, the UAV 110 communicates the captured videoto a satellite link and the satellite link provides the video to theserver 140, which may be ground based as an example.

The server 140 or another computer may transmit commands to the UAV 110for operating the UAV 110. The UAV may include a receiver (not shown)for receiving the commands. The server 140 or the other computer maytransmit commands to the UAV 110 that adjust its altitude, speed,heading, and its camera 130. The processor of the UAV 110 may executethe received commands.

In the course of its travels, the UAV 110 can be used to accumulate andtransmit vast amounts of UAV videos. However, it is difficult for ahuman operator to effectively scan through the overwhelming amount ofUAV videos to identify specific events of interest. Thus, according toat least one embodiment of the present invention, the server 140 isconfigured to automatically analyze the video content and selectivelycreate a concise summary and visualization that captures the mostinteresting events in the video.

In at least one exemplary embodiment of the invention, the events ofinterest consist of moving objects on the ground, along with theirtrajectories (e.g., a path that a moving object follows through space asa function of time). Examples of the object include people, vehicles(e.g., tanks, cars, trucks, etc.), animals, or complex objects such as aperson with a weapon, a person with a container, etc.

In at least one exemplary embodiment of the invention, the server 140executes a process that ranks event clips according to a specificsaliency metric, and provides a user-friendly visualization of a summaryof the key frames and video clips.

FIG. 2 illustrates a high-level block diagram of the process accordingto an exemplary embodiment of the invention.

The process performs a frame registration on several input video frames(S201). The frame registration may be performed by applying aforward/backward frame differencing step on the input video frames.

Based on the frame registration, the process performs feature-basedmatching (S202) and blob detection (S203). The feature-based matching isused to identify objects. The blob detection is used to preventartifacts in the videos that are not actual objects from beingsubsequently tracked. Morphological operations may be applied to theregistered frames for detecting potential blobs (i.e., the artifacts)and blob verification may be performed using an optical flow method.

The process then tracks multiple objects based on the identified objectsand the identified blobs (S204). For example, the identified objectsthat correspond to the identified blocks can be excluded from tracking.

After the moving objects have been detected and tracked, a saliencyscore is computed for each tracked object (hereinafter referred to as atrack), and the tracks are grouped into video segments and ranked(S205).

Then the process generates a summary video visualization based on therankings (S206). For example, the process may generate a single-imagerepresentation for each selected video segment in the form of akeyframe.

FIG. 3 illustrates an example of the single-image representation, whichincludes tracks A, B, C, D (e.g., moving people) and their respectivetrajectories. For example, as shown in FIG. 3, track A begins at thestart point of its associated trajectory curve at time 00.37 and reachesthe end point at time 00.58, track B begins at the start point of itsassociated trajectory curve at time 00.29 and reaches the end point attime 00.58, track C begins at the start point of its associatedtrajectory curve at 00.43 and reaches the end point at time 00.58, andtrack D begins at the start point of its associated trajectory curve at00.26 and reaches the end point at time 00.43. Thus, a single image canbe used to efficiently visualize movement of several tracks of interestwith respect to a particular background.

FIG. 4 illustrates a high-level flow chart for a method of detectingobjects to be tracked according to an exemplary embodiment of theinvention.

The method includes generating a series of frame alignments (S401). Inan exemplary embodiment, the generation of the frame alignmentscomprises computing the affine homography between two consecutiveframes. The affine homography may be computed by extracting Harriscorner points from both frames and then doing feature point matchingusing normalized cross correlation, which may be combined with a RandomSample Consensus (RANSAC) method for outlier removal. Affinetransformation parameters can be computed from as few as four featurepoints. The Affine transformation parameters can be determined using aleast squares technique, as an example. Once the affine homography iscomputed, it can be used to warp and align the frames so that temporalframe differencing can be performed.

However, instead of using simple subtraction from frame differencing, anexemplary embodiment of the invention uses a forward/backward framedifferencing method (S402). For example, a forward difference can becalculated from subtracting a frame of time t from a frame of time t+1and a backward difference can be calculated from subtracting a frame oftime t from the frame of time t−1, and then the resulting forward andbackward differences can be combined using the AND operator to generatea more selective difference image, from which objects can be detected.Use of the forward/backward differencing method may reduce motion noiseand compensate for illumination variation. Media filtering andthresholding can be performed on the calculated image (S403).Morphological operations can be performed on the result of the filteringand thresholding (S404). Blob filtering can then be performed on aresult of the operations to filter out false blobs (S405) to result inone or more objects of interest.

In an exemplary embodiment, to filter out the false blobs, an opticalflow based on sparse Kanade-Lucas-Tomasi (KLT) features is used toestimate the motion of each blob, and those blobs with no motion or amotion below a certain threshold can be discarded.

Once the objects have been detected, they need to be tracked over time.However, due to challenges in UAV video data, traditionalintensity-based target representation is no longer suitable due to largescale variation and perspective geometric distortion. Accordingly, aMeanShift tracking approach could be used as the main tracker. Toefficiently characterize the object, a histogram-based feature space maybe chosen. In an exemplary embodiment of the invention, metrics based ona Bhattacharyya coefficient may be used to define a similarity metricfor object tracking.

In an exemplary embodiment of the invention, to make tracking morerobust, a KLT based tracking approach is combined with the MeanShifttracking approach. A KLT based tracking approach is based on finding anaffine transform between objects from consecutive frames using therobust KLT feature matching, which is a similar technique to the framealignment step, but only applied to objects. The results from MeanShift,KLT and motion detection (e.g., results from forward/backward framealignment) can then be combined to better track the objects.

In an exemplary of the invention, a more precise tracking result isobtained by dynamically updating the target model for a tracked object.Both recent tracking results and previous target models may be used toupdate the current model, as formulated in the following equation 1:

q _(i) ^(t)=(1−α)q _(i) ^(t-1) +αp _(i) ^(t)  (1)

where q_(i) is the target model for object i, p_(i) ^(t) is the model ofobject i as detected at time t, and α is the learning rate (e.g., 0.1,0.2, etc.)

After the moving objects have been detected and tracked, a methodaccording to an exemplary embodiment of the invention is used to computea saliency score of each track. An object track T may be defined by asequence of 2D points in an image coordinate system [v₀, v₁, . . . ,v_(N)]. The saliency score of object track T may be computed accordingto the following equation 2:

$\begin{matrix}{{{Score}(T)} = {\frac{1}{N - 1}{\sum\limits_{N - 1}^{i = 1}\frac{1 - {\cos \; \alpha_{i}}}{2}}}} & (2)\end{matrix}$

where α_(i) is formed by each triple of points as shown in FIG. 5, and Nis the track length. According to equation 2, the score Score(T) isgreater or equal to 0 and less than or equal to 1. The higher thesaliency score, the smoother trajectory path of the tracked object. Inat least one embodiment of the invention, a higher saliency score ispreferred over a lower saliency score. For example, a track that followsa smooth trajectory path as compared to a jerky path is believed to morelikely correspond to an event of interest.

FIG. 6 shows a list of tracked objects sorted according to theirrespective scores. As shown in FIG. 6, the tracks that follow thesmoothest paths have the highest scores. For example, the first trackwith a score of 0.990817 follows the smoothest path while the last trackwith a score of 0.477574 is considered to follow the least smooth path.For example, the gait or pattern of movement of a person couldcorrespond to the path followed by the first track, whereas the pathfollowed by the last track may be too erratic to correspond to a person.Thus, a score threshold τ can be set, and then only the tracks with ascore above the score threshold τ can be selected (i.e., the mostsalient tracks), and the rest can be ignored.

Next, of the remaining tracks, groups are formed by those that can becontained by one single video segment. For example, FIG. 3 is an exampleof a group of 4 tracks (A-D) that can be contained by a commonbackground image over the course of a period of time from time 00.26 to00.58 (e.g., a video segment). For example, all four tracks are presentwithin the bounds of the background image for the entire period betweentime 00.26 to time 00.58. Further, it is assumed that all four track hadsaliency scores that exceeded the predefined score threshold τ. If thesaliency score of track A, for example, happened to be lower than thescore threshold τ, FIG. 3 would look similar, but would fail to includeTrack A and its corresponding Trajectory.

Each video segment having at least one salient track may be assigned itsown score based on the scores of the tracks it contains. For example,the video segment score for segment 00.26-00.58 could be determining bysumming the four saliency scores of tracks A-D. For example, if thetracks have respective scores such as the first four tracks in FIG. 6,the overall score for the video segment could be 3.852422.

The video segments having scores above a particular threshold can bepresented to an operator using a graphical user interface. Thesingle-frame image representation of a video segment (see FIG. 3) may bereferred to as a keyframe. FIG. 7 is an example of the graphical userinterface 701 that can present one or more of the keyframes in a panel702 and includes a menu 703 that enables the operator to select adifferent one of the available keyframes for presentation in the panel702. In the example shown in FIG. 7, the menu 703 lists 6 key framessorted according to their scores. Each item in the list indicates thestart frame (or a start time) at which the event corresponding to thekeyframe occurred and its length (or duration).

In an exemplary embodiment of the invention, the keyframe is created bysuperimposing all detected objects in the video segment into the firstframe of the segment. This may be accomplished by transforming eachtrack point in subsequent video frames back to the coordinate system ofthe first frame. In at least one embodiment of the invention, thetransformations are performed by using the homographies H between eachpair of frames, which have already been computed in the frame alignmentstep.

A transformation back 1 frame can be performed according to equation 3as follows:

F ₁ →F ₀ :p ₀ =H _(1→0) p ₁ =H _(0→1) ⁻¹ p ₁  (3),

and a transformation back t frames can be performed using a series of(3) according to equation 4 as follow:

$\begin{matrix} F_{t}arrow{F_{0}:{F_{t}\overset{{Ht}arrow{t - 1}}{arrow}{F_{t - 1}\overset{{{Ht} - 1}arrow{t - 2}}{arrow}\mspace{14mu} {\ldots \mspace{14mu} \overset{H_{1}arrow 0}{arrow}{F_{0}.}}}}}  & (4)\end{matrix}$

If a track point at frame t is back projected to frame will be outsidethe image boundary, in an exemplary embodiment of the invention, thetrack point is discarded.

FIG. 8 illustrates a high-level method according to an exemplaryembodiment of the invention for summarizing image content from videoimages (e.g., UAV videos) received from one or more moving cameras. Themethod includes detecting foreground objects in the images (S801),determining moving objects of interest from the foreground objects(S802), tracking the moving objects (S803) (e.g., using meanshift and/orKLT tracking), rating movements of the tracked objects (S804) (e.g.,smoother moving objects receive a higher rating, see above saliencymetrics), and generating a list of highly rated segments within theimages based on the ratings (S805). For example, the highly ratedsegments could be the segments that include several highly rated trackedobjects.

A summary display (e.g., FIG. 3, FIG. 7, FIG. 9, etc.) can be createdbased on the list of highly rated segments, where each highly ratedsegment is representative of a potential event of interest. For example,the summary display may display one of the highly rated segments and thehighly rated tracked objects it contains along with their trajectorieswithin a single image (e.g., see above discussion of keyframe).

The summary display may overlay the keyframe with display of thegeographic location of the start and end point of each of the displayedtrajectories. The summary display may overlay the keyframe with displayof the geographic boundaries or an approximate location of the keyframe.The summary display can overlay the keyframe with display of the startand end time of the video segment, with the identity of the camera(s)used to capture the images of the video segment, the identity of thevehicle that captured the images, the type of the vehicle, the type ofthe camera, etc.

The detection of the foreground objects may have been performed by aclassifier specially trained to find objects of a particular type (e.g.,military vehicles, military personnel, civilians, etc). Thus, thesummary display can overlay the keyframe with the detected type. Eachkeyframe may be referred to an event, and the summary display can beconfigured to convey the spatial organization of these events. Forexample, FIG. 9 is an illustration of a summary display where eachkeyframe is overlaid with a geographic map so the geographic location ofeach keyframe is clear. The summary display also illustrates thedate/time of each event, where the time may include the start and endtime.

The display may also include a number of knobs, such as space and timeknobs. The time knob may be used to adjust the period of interest sothat keyframes/events not present within the adjusted period may befiltered out and new keyframes/events present within the adjusted periodmay be filtered in. The space knobs can be used to adjust the mappresented so that keyframes/events not present within the boundaries ofthe map may be filtered out and new keyframes/events present within theadjusted map may be filtered in. The display can be configured such thatselection of a keyframe allows the user to zoom to see an expanded viewof that keyframe. The display may also include a toggle knob that allowsa user to toggle between viewing the events spatially as shown in FIG. 9and viewing the events in another format such as the one shown in FIG.7.

While embodiments of the invention have been described with respect to asingle UAV including one or more cameras, in alternate embodiments, oneor more additional UAVs may be present, and the image data from each maybe fused together to generate the resulting keyframes for display. Forexample, one UAV could be used to track objects in one portion of abackground and another UAV could be used to track objects in anotherportion of the background, and the highly rated tracked objects fromboth portions could be combined together into a single keyframe forpresentation to an operator.

FIG. 10 illustrates an example of a computer system, which may executeany of the above-described methods, according to exemplary embodimentsof the invention. For example, the methods of FIGS. 2, 4, and 8, may beimplemented in the form of a software application running on thecomputer system. Further, portions of the methods may be executed on onesuch computer system, while the other portions are executed on one ormore other such computer systems. For example, the object detection andtracking steps S801-S803 of FIG. 8 could be located on one computersystem while the rating and list generation steps S804-805 of FIG. 8could be located on another computer system. For example, an onboardprocessor of the UAV 110 itself could perform the object detectionand/or tracking. Examples of the computer system include a mainframe,personal computer (PC), a cable set-top box, a televisionmicroprocessor, a handheld computer, a server, etc. The softwareapplication may be stored on a computer readable media (such as harddisk drive memory 1008) locally accessible by the computer system andaccessible via a hard wired or wireless connection to a satellite or anetwork, for example, a local area network, or the Internet, etc.

The computer system referred to generally as system 1000 may include,for example, a central processing unit (CPU) 1001, random access memory(RAM) 1004, a printer interface 1010, a display unit 1011, a local areanetwork (LAN) data transmission controller 1005, a LAN interface 1006, anetwork controller 1003, an internal bus 1002, and one or more inputdevices 1009, for example, a keyboard, mouse etc. For example, thedisplay unit 1011 may be used to present the above described keyframes,summaries, etc. As shown, the system 1000 may be connected to a datastorage device, for example, a hard disk 1008 (e.g., a digital videorecorder), via a link 1007. CPU 1001 may be the computer processor thatperforms the above described methods.

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present disclosure may take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” ox “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readablestorage medium. A computer readable storage medium may be, for example,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium would include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), an optical fiber, a portable compact disc read-onlymemory (CD-ROM), an optical storage device, a magnetic storage device,or any suitable combination of the foregoing. In the context of thisdocument, a computer readable storage medium may be any tangible mediumthat can contain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable medium may be transmitted using anyappropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

1. A method for summarizing image content from video images receivedfrom a moving camera, the method comprising: detecting foregroundobjects in the images; determining moving objects of interest from theforeground objects; tracking the moving objects; rating movements of thetracked objects; and generating a list of highly rated segments withinthe video images based on the ratings.
 2. The method of claim 1, whereinthe rating comprises: determining a smoothness of a trajectory of eachtracked object; and assigned the tracked object a saliency metriccorresponding to the smoothness.
 3. The method of claim 2, wherein thesmoother the trajectory, the higher the saliency metric.
 4. The methodof claim 2, wherein the generating of the list comprises: discarding thetracked objects with a saliency metric below a threshold level; andgenerating groups of the remaining tracked objects that are containedwithin different respective time periods of the video images.
 5. Themethod of claim 4, further comprises: generating a score for each groupbased on the saliency metrics of the tracked objects of thecorresponding group; and discarding the groups whose score does notexceed a minimum threshold, wherein each remaining group corresponds toone of the highly rated segments.
 6. The method of claim 5, wherein thescore of a corresponding one of the groups is a sum of the saliencymetrics of tracked objects of the groups.
 7. The method of claim 4,further comprising: selecting one of the groups; generating a backgroundimage of the video images that contains all the tracked objects of theselected group; and overlaying the background image with the trackedobjects of the selected group.
 8. The method of claim 7, furthercomprising overlaying the background image with the trajectories of eachof the tracked objects.
 9. The method of claim 7, further comprisingsuperimposing the overlaid image over a geographic map that bounds thetracked objects.
 10. The method of claim 1, wherein the detecting of theforeground objects comprises detecting the objects from an aligned imagegenerated from performing image registration on consecutive frames ofthe video images.
 11. The method of claim 10, wherein the performing ofthe image registration comprises: generating a forward frame differencebetween consecutive first and second frames of the video images;generating a backward frame difference between the second frame and athird consecutive frame of the video images; and generating thealignment image from the differences.
 12. The method of claim 1, whereinthe tracking uses a combination of a KLT based tracking approach and aMeanShift tracking approach. 13-15. (canceled)
 16. A method ofpresenting a summary of a video, the method comprising tracking movingobjects within the video; determining a smoothness value of a trajectoryof each tracked object; discarding the tracked objects having smoothnessvalues below a minimum threshold smoothness level; selecting a period oftime within the video that includes at least one of the remainingtracked objects; and presenting a frame of the video overlaid with allof the remaining tracked objects that are present during the selectedperiod of time.
 17. The method of claim 16, wherein the selecting of theperiod comprises determining first and second periods of the video thateach include at least one of the remaining tracked objects; generating ascore for each period based on a sum of the smoothness values of thetracked objects present in the corresponding period; and selecting oneof the first and second periods having the highest score.
 18. The methodof claim 16, wherein the frame is the first frame of the video of theselected period.
 19. The method of claim 18, wherein the presentingcomprises transforming points of each track object back in the frames ofthe selected period subsequent to the first frame to a coordinate systemof the first frame.
 20. The method of claim 19, wherein the transformingis performed using homographies between each pair of the frames.
 21. Themethod of claim 16, wherein the tracking uses a combination of a KLTbased tracking approach and a MeanShift tracking approach.
 22. Themethod of claim 16, wherein the tracking comprises: detecting objectsfrom an aligned image generated from performing image registration onconsecutive frames of the video; and determining moving objects ofinterest from the foreground objects.
 23. The method of claim 22,wherein the performing of the image registration comprises: generating aforward frame difference between consecutive first and second frames ofthe video images; generating a backward frame difference between thesecond frame and a third consecutive frame of the video images; andgenerating the alignment image from the differences. 24-25. (canceled)